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PROBLEM SOLVING COURTS: 

AN EVALUATION GUIDE AND TEMPLATE 


INTRODUCTION 

The Canadian Council of Chief Judges commissioned the development of a template for the evaluation of problem 
solving courts in the spring of 2016 . 1 The objective was to provide greater standardization in the way in which 
evaluations of problem solving courts are conducted, but also to provide guidance to problem solving court 
administrators, teams, and stakeholders about how such evaluations can be conducted. Thus, the Centre for Criminal 
Justice Studies at the University of New Brunswick Saint John campus was tasked with developing a basic framework 
that would serve the common purposes of most problem solving courts with regards to their implementation and 
their impact assessment. The template included in this document was designed to be flexible in nature, as evaluations 
often need to be adjusted to the unique context of the program being evaluated, its resources, and specific goals and 
objectives. 

Prior to presentation of the template, relevant contextual information on the nature of program evaluation 
procedures in general is first presented, followed by a summary of procedures that have been used to evaluate 
problem solving courts in the scientific (i.e., peer-reviewed publications) and gray (i.e., non-peer reviewed technical 
reports) literatures, which informed the template development. A supplement to the current report will be developed 
to describe the results of a survey distributed to professionals working within problem solving court contexts. This 
survey inquired about priority areas of focus for process/implementation and outcome evaluations in the problem 
solving court context; that is, we asked the “knowledge users” what they would like to know about these types of 
court programs. If warranted, this supplement will also address additional problem solving court evaluation 
considerations that may not have been considered in the development of the current report. 


Program Evaluation: Basic Approaches and Considerations 


A program evaluation generally refers to the application of research methods to systematically evaluate the processes 
and impact of a program (Newcomer, Hatry, & Wholey, 2015 ). Although several guidelines and standards exist that 
describe best practices in program evaluation, there is no consensus on how evaluators should approach an evaluation 
(Stufflebeam & Shinkfield, 2007 ), particularly within a problem solving court context (Slinger & Roesch, 2010 ). 
Based on a review of the program evaluation literature, the points below outline several key issues which should be 
considered and discussed between the individuals requesting the evaluation (i.e., the court) and evaluators during 
the planning phase of an evaluation (Holden & Zimmerman, 2009 ) 

1. A court considering an evaluation should ensure that the evaluation is focused (Rossi, Lipsey, Freeman, 2004 ). 
As such, the court should ensure the questions to be answered by the evaluation are realistic and appropriate, 
that the evaluation is feasible and able to be completed in a reasonable amount of time, and that there are 
sufficient resources (e.g., access to data sources, administrative support) available to support the evaluators in 
successfully carrying out the evaluation. It can be helpful at this stage to consider the specific use of the 
evaluation results. For example, will the results be used to justify continued funding for the court, to 
understand the court’s processes to improve its service delivery and/or minimize operational costs, or to 
expand the program? 

2. The stage of the court’s lifecycle is an important consideration for program evaluation planning. The design 
and methodology of an evaluation will take on different forms depending on whether the court is in the pre¬ 
implementation stage, recently been implemented, or has been in operation for a period of time. More recently 
established courts may require more time to conduct an evaluation as it takes time for a sufficient number of 
clients to be processed through the court. Well-established courts may have years of data that could be more 


1 Throughout this report, we use the term “problem solving” court as an umbrella term to represent a designated court process that includes 
the use of a team of legal, mental health, correctional, and other professionals who work with a Judge to respond to criminal justice matters 
under the principle of therapeutic jurisprudence. These programs aim to meet the personal and social needs of program participants thought 
to underlie their criminal behaviour while also balancing public safety concerns and monitoring these cases through court appearances and 
case conferences. In some jurisdictions, these court programs are also referred to as “specialized” or “specialty” court programs and systems. 
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quickly accessed and analyzed and could potentially be evaluated in a shorter period of time depending on the 
nature of existing information. 

3. The relationship between the evaluators and the court should be clearly delineated. A court may choose to 
keep evaluators relatively independent, placing responsibility on the evaluators for all aspects of the evaluation 
planning and its execution. One advantage of this approach is that the results from independent evaluations 
are viewed as more valid (i.e., less prone to bias) by outside reviewers. In contrast, a court may choose to 
engage in a more collaborative partnership with the evaluators, in which the court is involved in the evaluation 
planning and execution. This approach may offer rich insights into the court’s operations that can be of great 
benefit to the evaluation and the interpretation of findings; however, there is potential that evaluation results 
may become more susceptible to bias when the evaluator is not viewed as truly independent. It is important 
to note that regardless of approach, evaluators by nature of their training and professional standards may 
always employ measures throughout the evaluation to maximize their objectivity. Independent evaluators 
should spend time observing the process of the program to obtain greater understanding of its nuances and 
operational context. 

4 . The type of evaluation the court requests has implications for the conclusions that can be made about the 
court’s operations and impact (Newcomer et al., 2015 ; Patton, 1997 ). Formative evaluations, also referred to 
as process or implementation evaluations, are conducted prior to or as part of a court’s implementation. In 
contrast, outcome evaluations are used to examine the court’s effectiveness or impact on short, intermediate, 
and long-term outcomes. In addition, an evaluation design may be cross-sectional, retrospective, or 
prospective in nature. Cross-sectional designs provide a snapshot of a court at a single point in time with less 
emphasis on how a court has performed in the past or may perform in the future. Retrospective designs are 
archival in nature as they look in the past to evaluate the court’s processes and outcomes up to the present or 
a specified date; however, findings may not reflect the current program if there have been substantial changes 
in the past to the program relative to what it looks like now in operation, resources, and policy. Prospective 
designs are ongoing studies that track current court processes and impact in real time and into the future over 
a period of time, anywhere from a few months to a couple years. Prospective designs are typically considered 
the most methodologically rigorous design and can better address questions regarding causal links between 
program activities and outcomes; however, they are complex and time-consuming. One final consideration is 
whether the court prefers to have quantitative or qualitative information, or both. Quantitative methods make 
use of numerical information that can be subjected to statistical analysis, usually focusing on administrative 
data, client file record information, and survey data. Qualitative methods are descriptive and make use of focus 
groups, individual interviews, or court/participants observations. Use of both quantitative and qualitative 
methods is often used and known as a mix-method approach. 

5 . Courts may be interested in knowing about whether their program is having a direct/causal impact on 
reducing and/or improving a given outcome (e.g., recidivism reduction, mental health recovery, cost-savings). 
The gold standard research design used to infer causality is the randomized controlled trial. 2 Unfortunately, 
due to a number of issues such as small sample sizes, resource limitations, logistical impracticalities, and ethical 
proscriptions, randomized controlled trials are rarely conducted in program evaluations (Newcomer et al., 
2015 ; Posavac, 2015 ). Both the court and evaluators alike must acknowledge that many variables, including 
variables outside of the program beyond anyone’s control, can influence an outcome. Thus, it becomes 
extremely difficult to conclude with certainty that the program itself was the sole casual agent in creating 
change in a given outcome, particularly in instances in which evaluators have to forgo methodologically 
sophisticated procedures in favor ofless rigorous procedures. Furthermore, evaluators may carefully word 
their conclusions to reflect the lack of causality, suggesting that a program appears to be associated with an 
impact rather than directly having an impact. Use of a control or comparison group that is matched on key 
characteristic with program participants (i.e., age, gender, ethnicity, risk level, needs, propensity to be 
admitted to the program) can mitigate some of the weaknesses of a non-random controlled trial when it is not 
feasible to randomly assign participants prospectively to the program. 


- In a randomized controlled trial study, people are randomly assigned to either a treatment group or a control group. The treatment group 
receives the program/service and control group does not receive the program/service, and both groups are compared on some outcome 
variable. 
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6. Potential sociopolitical issues that may impact the evaluation should be discussed. For example, societal 
attitudes in some jurisdictions (e.g., ‘tough on crime’ attitudes, stigma surrounding mental health and 
addiction) may be antithetical and unsupportive of the court. There is research that suggests public opinion 
plays a role, at least in part, on whether a court continues to receive funding (McDougall et al., 2012). In such 
a case, negative public attitudes may impact the evaluation by placing pressure on evaluators to produce 
positive outcomes, warranting a more independent role for the evaluator to minimize introducing bias into 
the evaluation. Another important consideration is to have an awareness of, and sensitivity towards, the 
sociocultural issues and systemic barriers that may have a direct and indirect impact on evaluations involving 
courts that provide services to First Nations individuals and marginalized groups. 

7. The program’s logic should be well described and understood by all parties. More formally known as logic 
models, these conceptual and visual frameworks describe how a program theoretically works and how the 
program is designed to achieve its goals and intended outcomes. Major aspects of a program are listed, which 
include inputs (i.e., resources that go into the program), activities (i.e., program process and procedures), 
outputs (i.e., the result or quantification of the activities), and outcomes (i.e., short, intermediate, long-term), 
usually with arrows depicting the relationships between events, such as a particular activity and outcome. 
Creating a well formulated logic model serves as a communication tool for all stakeholders and creates a useful 
guide in planning the evaluation. 


Problem Solving Court Evaluation Procedures: What Has Been Done Before? 


Current Context of Problem Solving Court Evaluations 

To inform our evaluation template, we conducted a review of the state of the research on problem solving courts. 
Our review considered evaluations of mental health courts, domestic violence courts, and drug courts given that 
these types of courts are more commonly represented in the literature, but also included available evaluations for 
community courts, veteran’s courts and health/wellness courts. Our review emphasized problem solving courts in 
Canada whenever possible, but also included problem solving courts from the United States, given the wider range 
of available data in that country. Both the peer-reviewed published literature and the non-peer reviewed “gray” 
literature were examined to obtain an understanding of the typical practices for evaluating problem solving courts. 
In this review, we made note of the evaluation methods used, the nature and source of collected information, and the 
types of criminal justice and non-criminal justice outcomes examined. Our review produced several observations 
pertaining to process/implementation and outcome evaluations in the context of problem solving courts, which are 
summarized below. 

Process/Implementation Components of Existing Problem Solving Court Evaluations 

Most problem solving court evaluations have focused on resource-related information relevant to the court’s 
operations, including the number of court appearances made by program participants, the type and number of staff 
involved with the court team, the nature of clinical and social services delivered as part of program involvement, the 
nature of sanctions delivered for non-compliance, and completion rates. Most evaluations report summary 
information about program participants, such as the number of referred, screened and admitted cases, as well as age, 
gender, ethnicity, and criminal history information (Rossman, Willison, Mallik-Kane, Kim, Debus-Sherill, & 
Downey, 2012). This information is then compared to program admission criteria to determine whether the program 
is reaching its target population. 

Samples sizes across problem solving court studies can vary greatly depending on the resources, location, and type 
of court. For example, an evaluation study of the Yukon community wellness court reported that the program had 
accepted 47 clients over the course of four years (Hornick, Kluz, Bertrand, 2011), whereas a Vancouver drug 
treatment court evaluation reported approximately 659 participants in their program over the course of 8 years 
(Somers, Rezansoff, & Moniruzzaman, 2014). 

Information about participants’ risk of reoffending is rarely formally assessed or reported as part of the admission 
screening process in these evaluations, making it difficult to know whether these courts are targeting low, moderate, 
or higher risk cases for criminal behaviour or specific forms of criminal behaviour. However, in recent years there 
has been a shift towards incorporating evidence-based recidivism risk assessment tools in drug courts. An evaluation 
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by the Department of Justice Canada (2015) found that drug courts in Edmonton, Ottawa, Regina, Vancouver and 
Winnipeg have implemented the use of risk assessment measures. Furthermore, Campbell and colleagues (2015) 
used a structured risk assessment tool to measure participants’ risk of recidivism in a mental health court setting. 

Occasionally, these evaluations report the amount of time passed (usually in days) from referral to admission decision, 
as well as the time from program entry to discharge to learn about resource needs and time efficiencies in the 
program. Average time from program entry to exit varies considerably depending on whether the individual 
graduates or is terminated prematurely (the former is usually in the program for a longer period of time) and on the 
type of court given that each will have their own completion criteria. Although completion (or graduation rates) are 
commonly recorded, explicit definitions of what is meant by completion is often not provided or clearly 
operationalized. Some evaluations provide detailed information about the reason for discharge from the program in 
cases of non-completion, which is useful for providing some context as to why some cases do not complete the 
program (i.e., voluntary withdrawal vs expulsion due to noncompliance; Hoffart, 2011). 

Outcome Components of Existing Problem Solving Court Evaluations 

For most problem solving courts, the primary outcome tends to be criminal justice orientated. Specifically, one of 
the most common outcome indices are recidivism rates which have been most frequently captured by measuring the 
number of new arrests/charges or time to new arrest/charge since program admission or since program discharge 
over a specified follow-up period. Based on the literature, most courts will follow their participants for up to 12 
months post-completion and record new offenses incurred during this time period. However, some evaluations 
(mostly drug courts) will follow participants for up to three years. One research report on New York’s drug courts 
tracked participants for 1, 2, and 3 years and found a significant impact on the reduction of re-arrest after the 3 years 
(Cissner et al., 2013). Recidivism rates are typically compared to: a) program non-completers, b) individuals who 
decided not to participate in the court or who were not admitted after referral, or c) individuals who received criminal 
justice and correctional “treatment as usual” (i.e., processed through traditional court). The latter comparison group 
is the least commonly used despite being the most useful for demonstrating unique program outcomes beyond the 
usual criminal justice system response when members of this group are similar in key characteristics to problem 
solving court participants, or when randomly assigned to problem solving court vs treatment as usual conditions 
(Cosden, Ellens, Schnell, & Yamini-Diouf, 2005; Lowder, Desmarais, & Baucom, 2015). 

Beyond criminal justice outcomes, examples of more specific outcomes evaluated depend on the goals and focus of 
the individual problem solving court. For example, drug court evaluations sometimes examine changes in 
participants’ addiction status and their substance use at the end of the program (Conley, Allen-Blakney, & Stoeckel, 
2013). Mental health courts tend to focus on improved mental health functioning, emergency department/crisis 
service use, hospitalization, and addiction status as well (Campbell et al., 2015). Lastly, domestic violence courts 
examine factors associated with reduced violence in intimate relationships (Cissner, Labriola, & Rempel, 2015). 

Problem solving court evaluations have also examined broader quality of life outcomes that can directly or indirectly 
influence criminal behaviour risk, such as changes in employment status, education, and housing stability. Problem 
solving courts that require participants to obtain employment and permanent housing tend to have a higher success 
rate in terms of re-offending reduction than those that do not, presumably because of the stabilizing influence of 
these factors (Pulsipher, Dyer, & Fuller, 2015). 

Challenges with Existing Problem Solving Court Evaluations 

The challenges problem solving courts face can range from maintaining necessary financial backing to high drop¬ 
out rates. In addition, some courts do not have the necessary resources that would allow them to gather and record 
data accurately to evaluate their programs, especially when these programs were designed without consideration of 
program evaluation requirements. Furthermore, most evaluations of problem solving courts fail to record the nature 
of interventions being used to achieve its goals, whether that be crime reduction, enhanced mental health, or 
addiction recovery. Thus, it is difficult to gauge whether evidence-based interventions are being used as part of the 
program model itself or through its community partners delivering these services. To maximize the success of any 
problem solving court, the methods used to change the behaviour of its participants and to enhance their prosocial 
actions and life choices requires use of evidence-informed intervention strategies, and testing whether this has been 
the case has not often been a priority of existing evaluations for problem solving courts. Lastly, an overreliance on 
pre-post designs in the absence of randomly assigned control groups or matched comparison groups made up of 
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individuals serviced in traditional criminal justice contexts limits the methodological rigor of a problem solving 
court evaluation. 

Pre-post designs compare a person at baseline to one or more follow-ups (e.g., at discharge or one year post¬ 
discharge) to determine whether change occurred on certain characteristics. However, without the inclusion of a 
control or comparison group, it is not possible to attribute these changes solely to participating in the program. 
Comparison groups typically consist of individuals who were referred and not admitted to the program or who 
prematurely left the program. These types of comparison groups have their limitations, namely that individuals who 
were referred but not admitted or who leave early may differ in fundamental ways from those who remain in the 
program (e.g., higher risk, more complex mental health issues). Thus, without random assignment, proper statistical 
adjustments or matching procedures on key characteristics, any detected differences between groups may have more 
to do with pre-existing differences than to the program being evaluated. However, randomized controlled trials, 
although rare, do exist. For example, the Oregon Criminal Justice Commission and NPC Research group (2015) 
conducted a multi-site drug court evaluation that randomly assigned participants to the treatment group (i.e., the 
drug court program) or a control group who received treatment as usual (i.e., traditional court processing) and were 
compared on several outcomes such as recidivism, program completion, and characteristics predicting program 
success. In addition to a paucity of randomized controlled studies, most problem-solving courts evaluations are not 
long term studies. Therefore, this makes it difficult to assess the long-term benefits of problem-solving courts. 


A Problem Solving Court Evaluation Template 


The template articulated in this report contains elements relevant to both process/implementation evaluation and 
outcome evaluation. The specific data included in an evaluation will vary depending on each court’s individual 
program goals and intended outcomes, as well as their unique evaluation goals. The source of this data and method 
of measurement is informed by these factors. Thus, to frame a standardized evaluation template, we needed to make 
specific decisions about the intended program goals and evaluation goals. We selected goals that would likely be 
common to most problem solving courts, but recognize that additional components or methods may be added to 
address more nuanced and complex evaluation questions (e.g., cost-benefit analysis, ethnographic analysis of team 
decision-making processes). Thus, for this generic evaluation template, we assumed the following goals: 

• Assumed Program Goals 

1. To reduce the frequency of criminal justice system contact; reduce the severity and volume of criminal 
behaviour; increase the amount of time elapsed between offending incidents; 

2. Other goals may be added here that are specific to the individual problem solving court goals. 3 

• Assumed Evaluation Goals 

1. Process evaluation goals 

a) To determine the degree to which the established policies and operational protocols for referral, 
admission screening, program planning, and discharge processes function as expected and are 
appropriately resourced; 

b) To learn of the program operational experience from its participants and stakeholders involved 
with the program; 

c) To provide recommendations for building on program strengths and for modifying areas needing 
improvement. 

2. Outcome evaluation goals 

a) To determine the degree of reduction in criminal behaviour and criminal justice contacts achieved 
relative to the 12-month period prior to problem solving court referral, the period during problem 


3 For example, mental health courts would focus on enhanced mental health recovery indicator goals; drug courts would focus on addiction 
recovery indicator goals; and domestic violence courts would focus on reductions in domestic or intimate partner violence, risk factors for this 
form of violence, and enhanced protective factors against the use of aggression. Regardless of the type of problem solving court, these personal, 
social and/or clinical change goals should be operationalized (i.e., defining concepts into a measurable and quantifiable index that can be 
measured) to inform the method of measurement used to assess them as outcomes. Consultation with experts in these specialized areas is 
strongly recommended to ensure that program goals target factors most directly associated with reductions in the targeted form(s) of criminal 
behaviour and the population of focus for the court program. 
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solving court involvement (can vary but usually 12-16 months on average), and the 12-month 
period following discharge from the court for program participants; 4 

b) To determine the degree of reduction in criminal behaviour and criminal justice contacts relative 
to a similar group of individuals eligible for admission who did not participate in the problem 
solving court (e.g., waitlist control group or criminal justice system response as usual comparison 
group); 

c) To learn of the program outcomes from its participants and stakeholders involved with the 
program. 

d) Other goals as dictated by specific targets for change of an individual problem solving court, such 
as mental health recovery, addiction recovery, housing stability, procriminal attitudes, service 
access improvements, etc. 

e) To provide recommendations for building on program strengths and for modifying areas needing 
improvement. 

Although we have included generic problem solving court evaluation components, the template described below is 
flexible in its application and can be adapted based on resources and timeframes available to conduct the evaluation. 
A logic model of a generic problem solving court evaluation framework is included in Appendix B, which provides a 
visual representation of comprehensive evaluation content. This model guides evaluators to consider the inputs 
going into the program that are required for it to be operational, the activities that form the body of the program, 
the outputs that occur as a result of these activities, and the short-, intermediate, and long-term outcomes achieved 
as a result of these outputs. Collectively, this logic model guides the focus of the evaluation depending on which 
aspects of the program model are being prioritized for evaluation. Although we have included reference to cost- 
savings/cost-benefit analyses, it is beyond the scope of this template to include procedures necessary for these types 
of calculation and they are not discussed further. Consultation with experts in this specific form of analyses is 
recommended if this is a goal of the evaluation. 

Process/Implementation Generic Evaluation Components 

One of the major objectives of a process evaluation is to determine whether the program is being implemented as it 
was designed to be implemented. In the context of a problem solving court, basic questions that should be addressed 
at this stage of an evaluation include: a) is the court targeting or reaching the intended people? b) Is the problem 
solving court’s procedures and policies for referral, admission, case processing/management, sanctions, and 
discharge being followed as they should? c) Is the problem solving court sufficiently resourced to achieve its goals 
in terms of staffing complement, staff training needs, physical space needs, equipment/material needs, and required 
services? Procedures for gathering information that will inform each of these questions are described below. Under 
ideal circumstances, these evaluation procedures should be embedded into the program as it is being designed to 
facilitate information gathering either in real time (i.e., prospective designs), or at a later date by going back into 
these records to extract the necessary information (i.e., retrospective designs). Evaluators can do more with these 
types of records than with records that may not have been developed or maintained with a mind to eventual 
evaluation, as the type of information or its quality may not be sufficient to adequately inform the evaluation without 
collecting new data. 

A. Recording of Referral, Admission, and Case Plan Information 

• Basic information needs to be available regarding the number of individuals who were referred to the program, 
the number of individuals admitted to the program once referred, and the number of people who complete or 
leave the program prematurely. It is from this information that essential details about the program’s rates of 
referral, admission, completion, and drop-out can be obtained. 

• Designing a standard Intake/Screening Form can assist with this type of record keeping regardless of 
whether the evaluation is being built-into the program upfront or will be conducted at a later date. At a 
minimum, this referral/intake form should include the candidate’s demographic information (age, gender, 
ethnicity) and referral information, including the type of criminal offence(s) attached to the referral and the 


■* Time frames of 12 months are usually sufficient to detect changes in most forms of behaviour. These time frames can be shortened or 
lengthened as needed, but should be consistent across periods of evaluation. It should be noted that evaluations with shorter time frames (e.g., 
6 months or less) may only be able to detect modest changes across social, personal, clinical, or criminal justice domains. 
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justification for the referral (which will vary by problem solving court, but could include presence of a serious 
mental illness, charge related to domestic violence, substance abuse concerns, etc.). 

• Depending on the resources available to the problem solving court, and its screening procedures, it is also 
helpful to include on the Intake/Screening Form information about each candidate’s risk of reoffending using 
standardized risk assessment instruments appropriate for the target population (e.g., Level of Service/Case 
Management Inventory for adult general recidivism or the Youth Level of Service/Case Management 
Inventory for youth general recidivism). Knowing the candidate’s risk of general (or violent) recidivism risk 
will provide valuable information as to the risk level of the individuals being referred and subsequently 
admitted to the program, especially if one of the program’s objectives is to work with “low risk” cases vs. “high 
risk” to reoffend cases. Similarly, the Intake/Screening Form should also include summary information with 
regard to the criminogenic needs (i.e., direct risk factors associated with criminal behaviour) and non- 
criminogenic needs (i.e., areas of concern but which may not directly impact criminal behaviour) of the 
candidate, as these pieces of information will provide rich information about targets for intervention to reduce 
criminal behaviour and to enhance client well-being, if this information is relevant to the goals of the problem 
solving court. Knowing risk information at intake informs the evaluation with regard to whether the 
operational procedures of the program function equally well for individuals at low, moderate and higher risk 
for recidivism. This information also can be used by evaluators to determine whether the program works 
equally well for individuals at all levels of risk for re-offending, or whether it is more impactful for individuals 
of lower versus higher risk levels. In turn, this information can be linked to a consideration of the 
appropriateness of the eligibility criteria and/or the level of services/supervision provided by the program to 
individuals of differing risk levels. 

• Another advantage of using an Intake/Screening Form to record information is that it provides a place to 
record the reasons for not admitting an individual to the problem solving court — that is, who is being excluded 
and why. Using a standardized checklist of reasons on the referral/intake form that are specific to the given 
problem solving court (e.g., does not meet criteria “x” for admission) helps maintain the consistency of data 
recording across parties being referred and across professionals recording the information. 

• Once the candidate has been admitted to the program, a Case Plan/Progress Form should be generated to 
keep track of each client’s progress and the nature of the services received as part of their involvement in the 
problem solving court through its own resources or its affiliated resources, if applicable. Specifically, this form 
should include the dates of court appearances, presence of new arrests/charges/convictions, sanctions used, 
types of services received from the problem solving court team or community/institutional resources, and a 
rating scale capturing perception of the client’s progress at each court appearance or after team case conference 
by a case manager or as a consensus rating by the team. 

• It is also useful to track the reasons for discharge from the program, especially when individuals leave the 
program for reasons other than program completion. A Discharge Form should be designed to capture this 
information in a consistent and standard manner across cases. At a minimum, this form should include whether 
the individual was discharged because of program completion/meeting program goals; voluntary withdrawal 
without completing the program; forced withdraw due to new charges; forced withdraw due to non-compliance 
with program plan/rules/expectations; or discharge for other potential reasons specific to the problem solving 
court and its policies for early discharge. Such information also speaks to the success of the program with 
meeting its targets with regard to the number of individuals who are screened, admitted, and complete the 
program. This information can be further tied back to the characteristics of the program participants captured 
in the Intake/Screening Form to understand which clients complete the program and which do not, and 
identify potential areas of strength in the program or areas for potential change in the referral, admission, and 
case planning/monitoring procedures and policies. 

B. Recording Operational Procedure Information 

• A process evaluation should provide information about whether procedural aspects of the program were 
followed or achieved as expected. Thus, it is necessary to record operational information about the problem 
solving court; doing so in real-time will save time and effort later on when attempting to compile this 
information. This procedural information will speak to resource needs as well as to intended problem solving 
court protocol adherence. At a basic level, it is helpful to record the following operational information in a 
spreadsheet: 
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> Dates of key procedures being implemented or completed so that evaluators can capture time 
responsiveness regarding the number of days passed between problem solving court referral to 
screening initiation, then to admission decision and to discharge from the program. 

> Dates and number of problem solving court sittings (i.e., to calculate a per year or per month 
occurrence average), and who the Judge was that presided over these proceedings (same or different 
Judges). 

> Dates of court appearances per client to calculate the total number and average number of 
appearances for program participants. 

> Dates of pre-court case conferences and who attended these case conferences (or at least what 
organizations they represented) to calculate the total number and average number of conferences 
held per participant and overall on a per month or per annual basis. 

> Nature and type of community agencies providing services to problem solving court participants. 

> Dates and number of problem solving court team meetings, separate from case conferences, in which 
operational or business matters are discussed. 

> Nature of training received by problem solving court staff/supporting professionals, number of 
training events, and/or number of training days. 

C. Recording Stakeholders’ Experience of the Operational Process 

• When examining the process and operational aspects of a program, it is helpful to gather contextual 
information from the professionals involved in delivering or supporting the program, as well as from those 
participating in the program (i.e., the clients). The information gathered from these sources provides a 
knowledge user and a participant perspective on the program and how it operates, which helps the evaluator 
understand the numerical data also being collected. A survey designed for the specific problem solving court 
being evaluated can be used to gather this information by sending it to key stakeholders (e.g., court team 
members, family members, partnering service delivery representatives, victim service representatives) and by 
asking program participants to complete it at various stages of the program (intake, midpoint and discharge) 
and at a pre-determined post-program discharge follow-up. 

• Another informative way to learn about the nuances of these perspectives is to conduct semi-structured 
interviews with stakeholders representing various roles in the program and with the client themselves. A semi- 
structured interview asks specific questions of interest to the evaluator, but allows for additional questioning 
based on the responses received. Focus groups also can be used for larger groups of community stakeholders 
to gauge their perspective on how the operations and expectations of the court work within the context of the 
services they provide to it and/or to the program participants. A sample semi-structured interview that could 
be used at the time of program discharge or post-discharge follow-up with program participants is provided 
in Appendix D, and a stakeholder interview or focus group question structure is found in Appendix E. The 
content of these interviews can be then analyzed for common and unique themes across interviewees that 
reflect their experiences. 

• For process evaluations, surveys and interviews should focus on the informant’s experience of each major 
phase of the program (referral, admission screening, program delivery, and discharge), with specific focus on 
what they found worked well and what worked less well about these processes and procedures. Program 
strengths can thus be identified, as well as reflect potential inefficient, confusing, or incorrectly implemented 
policies/procedures. If using interviews, they should be conducted by someone else other than the evaluators 
directly involved with the problem solving court or a supervisor/manager of the interviewee to maximize the 
opportunity for honest and free expression without fear of repercussions by one’s employer, case manager, or 
other person with authority over the interviewee. Ensure the confidentiality of the interviewee’s identity prior 
to commencing the interview, or be clear about the limits to this confidentiality in advance. 

• Whether administered in survey or interview form, a measure of participant satisfaction also should be 
obtained — this could be as simple as a rating on a scale of 0 (not at all satisfied) to 10 (completely satisfied) or 
could be ratings of satisfaction across various domains of interest pertaining to the operational procedures of 
the court program (e.g., separate satisfaction ratings for referral, admission screening, case 
management/services, and discharge processes) and/or pertaining to the quality of interactions with the 
program solving court team in general or with individual members/roles on the team. 
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Outcome Generic Evaluation Components 

A. Selecting and Defining Outcomes 

• An evaluator first needs to decide which outcomes they want to examine, which is best achieved when 
designing the problem solving court so that the necessary data can be obtained as participants progress 
through the various stages of the program. Consultation with problem solving court representatives is 
necessary to appreciate the intended outcomes of the specific problem solving court being evaluated. 
Construction and review of the program logic model will help evaluators identify and narrow the range of key 
outcomes of interest for measurement in the evaluation. 

• Once these outcomes have been decided upon with the relevant stakeholders, they should be operationalized 
so that they can be measured and recorded in some manner. 

> For example, “reduced criminalization” can be defined to mean a number of different things — reduced 
police contacts, fewer arrests/charges/convictions; reduced days incarcerated; fewer court 
appearances, etc. Thus, it is essential that these outcomes be concretely defined to ensure that they 
are measured correctly and represent the actual goals of the program. 

> Recidivism (i.e., new arrest, charge, or conviction) is perhaps one of the most common outcomes in 
the problem solving court literature. However, there is no consensus among researchers as to which 
is the most preferred index of recidivism. Within the broader offender recidivism prediction 
literature, most researchers view charges as the best index of offending behaviour patterns in the 
community. This is because charges are less confounded by errors of false arrests and are less affected 
by various criminal justice processes (e.g., plea bargains, withdraws/dismissals) as in the case of 
convictions. The appropriateness of using arrests, charges, or convictions depends on the design of 
the evaluation. For example, one important consideration when choosing a recidivism metric is the 
timeline of the evaluation as conviction data is less useful for evaluations with short follow-up 
periods, given the lengthy period of time required to get a conviction. 

> Although criminal justice outcomes are common points of interest across problem solving courts, 
evaluators will likely need to select other more program-specific outcome goals as well. These goals 
will depend on the specific program goals, and methods used to achieve those goals. For example, 
mental health courts should include outcome goals specific to mental health recovery indicators; 
drug courts to addiction recovery indicators; and domestic violence courts to risk factors associated 
with domestic/intimate partner violence. 

• Once concretely defined, an evaluator needs to decide how best to measure these outcomes. 

B. Choosing Outcomes Measures - Questionnaires 

• The selection of surveys or questionnaires depends on the specific outcome goals selected by the program as 
relevant to their work and the impact of the program on its clients, such as changes in self-reported antisocial 
behaviour, procriminal attitudes, acquisition of specific knowledge or skills, mental health and addiction 
recovery indicators, etc. It is beyond the scope of this template to provide a summary of all possible outcome 
survey measures, but consultation with professionals with expertise in measure selection and an understanding 
in psychometric analyses would be helpful for selecting the most appropriate instruments to capture change 
indices (i.e., supported by research as both reliable and valid). These measures can be administered during the 
screening phase, during the problem solving court participation, and time of discharge, and post-program 
follow-up to capture progress and change in the domains of interest. Depending on available resources and 
measures selected, these questionnaires can be completed by the program participant (i.e., self-report), by those 
knowledgeable of the participant’s progress (i.e., informant/observer report), and/or can be administered by a 
professional. This multi-informant perspective on outcomes provides a rich source of data for evaluators to 
speak to program outcomes. 

C. Choosing Outcomes Measures - Interviews 

• In addition to process relevant content, Appendix D and E semi-structured interview guides contain questions 
that inform understandings of program outcome from participant and stakeholder perspectives. 

> It is useful to interview participants of the program, including those who have not successfully 
completed it, to understand their experience of the impact of the program on their criminal behaviour 


9 I P a g e 


risk and their general life functioning, or other specific aspects associated with program goals. These 
questions should be balanced with regard to perceived positive, neutral, and negative impacts. In 
particular, when constructed appropriately, without the use of leading or biased questions, 
interviews with current or former program participants can provide a rich source of information 
about the various ways a program was helpful to them and in what ways it was not helpful, or was 
even harmful. Unintended negative consequences sometimes arise from program involvement (e.g., 
increased monitoring by police may lead to increased charges for breaching conditions relative to 
similar individuals in the traditional criminal justice system; experience of negative stigma attached 
to being a program participant), and it is important to be aware of these issues when interpreting 
the outcomes from an evaluation. 

> Interviewing professionals who contribute to the problem solving court, or who provide supports 
and services to its clients, can provide valuable insights from an observer perspective as to the nature 
of any gains that result from the problem solving court’s work. These gains could include facilitated 
information sharing, faster access to services/supports for clients, greater collaboration between 
service agencies and community organizations, as well as gains for client outcomes and public safety. 
These interviews also highlight whether gains are not occurring as expected, and highlight where 
detrimental outcomes have occurred for program participants, service systems, and public safety. 

> These qualitative forms of data can provide the needed context in which to fully interpret and 
understand questionnaire/survey outcome data and the information gathered from case records 
about outcomes resulting from the problem solving court. Common and unique themes are identified 
by reviewing interview data, often from written transcripts of these interviews and/or from notes 
maintained by the interviewer while the interview was being conducted. It is helpful to have at least 
two individuals review the interview material to minimize bias in its interpretation and to ensure 
that others perceive similar themes in the interview content. These interviews should be conducted 
by a person(s) trained in interviewing skills, and who is not a direct service provider/case manager 
to the program participant being interviewed. Likewise, this interviewer should not be a supervisor 
or manager of the professional being interviewed. Being careful about these ethical considerations 
will encourage interviewees to speak more freely and honestly about their perception of program 
outcomes without fear of repercussions or disappointing the interviewer. 

D. Choosing Outcomes Measures - Records Review 

• A common source of information used in program evaluations are records maintained as part of the program. 
Depending on the nature of the recorded information and the quality and integrity of this information, much 
information can be extracted about a program’s operations as well as its outcomes. When an evaluation is 
integrated into the operational goals of a program, the file information is maintained in such a way as to meet 
operational goals and evaluation goals. This is the ideal situation, and saves much time later when it comes 
time to extract and use this information for mid-point or final evaluation reports. The use of client 
screening/intake, case processing/management, and discharge forms facilitates the gathering of the desired 
information in real time by program staff or trained research assistants/volunteers as clients are processed 
through the problem solving court. A summary of the recorded data typically used in problem solving court 
evaluations can be found in the Logic Model in Appendix B under activities and outputs for process focused 
evaluations and in the short-, intermediate- and long-term outcomes section for outcome evaluations, as well 
as in the Sample Data Collection Forms displayed in Appendix C. Additional outcome data may be added to 
reflect unique program outcome goals of an individual problem solving court. Additional records that should 
be made available to the evaluator are documents pertaining to the program model description, operational 
policies/protocols, forms used as part of the administration of the program (i.e., referral forms, etc.), and 
sometimes meeting minutes for operational aspects of the program. 

> To maintain the integrity and usefulness of file information, the recorder of the information must be 
very clear as to what information is to be captured and how this information is operationalized (i.e., 
defining concepts into a measurable and quantifiable index that can be measured). Specifically, two 
people recording this information should have the exact same understanding of what is supposed to 
be recorded and how. When differences of opinion and interpretation occur in how and what data is 
to be recorded, this mismatch then compromises the integrity of the data and limits its utility in the 
evaluation. These differences in opinion can be resolved by holding a meeting between the two 
coders to discuss and clarify the definitions/guidelines as to how to record the information. 
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> Capturing this data in an electronic record system that automatically uploads the information into a 
database is the most efficient way to gather case processing information, especially in the absence of 
research assistants or staff with time to extract the information from records into a database for 
analytic purposes. If such a system is not feasible, then a staff member or research assistant or the 
evaluator will need to go through paper and/or electronic records in which information is recorded 
about referred cases, screened and admitted cases, case planning/management information and 
discharge records. If these records are not held in a centralized problem solving court file, or these 
records are incomplete, then the evaluator may need to access records held by other organizations, 
such as mental health services, social services, and correctional services. Appropriate informed client 
(guardian) consent to access this information for evaluation purposes is typically required; however, 
it is possible to seek permission from institutions holding these records to waive informed consent 
through the application of secondary use data protocols specific to provincial and federal privacy 
legislation and the policies of each of these individual organizations. This process can become 
complex and time consuming to work through; thus, it is much easier and time efficient to obtain 
program participants’ permission at the time of referral to the program to access their health, social 
services, correctional, or other records not maintained specifically by the problem solving court, but 
of relevance to the assessment of evaluation outcome goals. 

E. Selecting an Outcome Research Design 

• Which ever measures are used to capture outcome data, an evaluator must decide on the most appropriate 
research design to assess change and outcomes within a particular problem solving court. 

> Option 1 - Pre-Post Designs: Most existing problem solving court evaluations have used a pre¬ 
post research design which follows program participants on various metrics over time to determine 
whether and how they changed on key factors relevant to the goals of the problem solving court. 
Pre-post information is valuable as it helps evaluators understand in what ways program 
participants’ may have changed as a result of being part of the program, and also informs an 
understanding as to why some participants may not have completed the program. If formalized 
measures of change have been included in the Intake/Screening procedures of the problem solving 
court, then these also should be recorded on the Discharge Form. Change metrics might include a 
re-assessment of recidivism risk, mental health measures, domestic violence measures, substance use 
measures, or changes in the rate/severity of criminal behaviour, etc. specific to program goals and 
objectives. Preferably, the same measures administered at intake should be used at discharge to allow 
direct comparisons from pre-program to discharge time points. Use of a post-program follow-up 
period during which similar measures are re-administered is also informative for determining 
whether change is sustained, enhanced, or weakened after leaving the problem solving court 
program — most evaluations have used 12-24-month post-discharge follow-up periods to speak to at 
least short to medium-term impact. If using self-report questionnaires or interviews to collect 
change information, then post-program follow-up requires sufficient evaluation resources to track 
clients and meet with them to gather the necessary information when they are no longer part of the 
program. 

r Option 2 - Comparison Groups Designs: When resources permit, use of a comparison group will 
add value to the evaluation. When carefully constructed, a comparison group will allow evaluators 
to determine whether individuals similar to the ones admitted to the problem solving court in 
question experience the same or different outcomes. Such a broader context allows an evaluator to 
state whether the problem solving court program led to enhanced, similar, or worse outcomes than 
found among similar individuals who received traditional criminal justice processing. The key to 
selecting a comparison group is to ensure that it lias similar baseline/intake characteristics to that 
of the problem solving court participants, otherwise any differences between these two groups at 
outcome might have more to do with these pre-group differences than to the systems by which their 
legal matters were addressed. The hallmark research procedure to maximize equivalence between 
groups at intake is to randomly assign potential, eligible participants to service-as-usual or to the 
problem solving court. This option is not always possible on an ethical or practical level, and thus 
an alternative is to match problem solving participants with service-as-usual participants on key 
characteristics and only include those matched individuals in the comparison group. Matched 
characteristics often include age, gender, ethnicity, criminal risk, mental health or social service 


11 IPage 



needs, propensity to be admitted to the program, or other characteristics that may be relevant 
depending on the specific problem solving court goals. Statistical procedures can then compare the 
comparison and problem solving groups after the matching procedure to confirm that these two 
groups are similar on key characteristics prior to program initiation. In this way, evaluators can have 
more confidence in the outcomes reported in comparison to what happens to individuals eligible for 
the program but who were processed by other means. A third alternative is one in which pre-group 
differences are statistically “adjusted” or “controlled” as part of the data analytic procedure. This last 
option is the least preferred option but still appropriate in situations where random assignment and 
matching procedures are not feasible. 

Ethical Considerations 

Collecting information about participants has implications for respecting their privacy and their rights. If an 
evaluation is embedded into the design of the problem solving court upfront, then potential program participants 
should be informed that the information they provide to the problem solving court may be used for the purposes of 
evaluating that program and that, by consenting to the program, they are consenting to the use of their information 
in the course of the evaluation as well. Alternatively, some evaluations give program participants the option to 
separately consent to having their information used in the evaluation; that is, they agree to be considered for the 
program or participate in the program, but may not want to be part of the evaluation. Consent for participating in 
the evaluation can be included in the body of text used to obtain informed consent or assent for program participation 
if a voluntary referral/admission process is being used. Mandated court programs should still consider whether 
informed consent to use the individual’s information in the evaluation is required based on ethical principles and 
privacy legislation relevant to their jurisdiction. 

These consent forms should be specific as to the nature of information to be accessed, how it will be used in general, 
and who will have access to the information, as well as the right of the participant to refuse consent for sharing of 
information for evaluation purposes. In cases of incompetency, then a legal guardian should be substituted, though 
even in these cases it is ethical to obtain client assent if possible. Secondary use data (i.e., using information initially 
collected for program purposes later in an evaluation, such as in the case of a file review evaluation process) does not 
always require informed consent from individuals about whom the information relates (i.e., clients), especially if that 
original information is de-identified 5 . Specific guidelines for informed consent, use of secondary use data, and waiver 
of informed consent conditions are provided in Canada’s Tri-Council Policy Statement on the Ethical Conduct for 
Research Involving Humans (see http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2- 
eptc2/Pefault/ ). Such information is typically required when evaluation plans are submitted to human ethics 
research boards and/or privacy officers for approval prior to the start of an evaluation. 

A sample informed consent form for a problem solving court evaluation is contained in Appendix F. This consent 
form is relevant to situations in which consent to collect information for the evaluation is being obtained at the time 
of program referral, eligibility screening, or even at the time of program admission. Variations of this content could 
be used for participants currently in the program, or who have left it and follow-up information is being sought. A 
similar styled consent form could be used with comparison groups who are not referred to the program, as the goals 
of the evaluation remain the same and the tasks required of the participant are likely similar. 

Data Storage and Security 

As with any confidential record, data collected for the purposes of the evaluation should be stored safety and securely, 
with access restricted only to those individuals directly involved in the evaluation. Paper records should be stored 
under lock and key in secure areas, and electronic records should be password protected and stored on encrypted 
devices that are securely stored when not in use. Care also should be taken not to identify individual evaluation 
participants when reporting evaluation findings. Data should be reported only in a de-identified (no names, etc.) 
manner that respects the confidentiality of these data and the individuals who provided it. 


5 De-identified data refers to information that has been recorded, but contains no personal identifying information such as 
names, addresses, places of work, dates of birth, etc. These data typically separate individual cases by use of a randomly 
generated identification number to identify the case information. 
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Required Resources 

When planning an evaluation, careful consideration should be given to the resources needed to conduct it. 

Specifically, who should conduct the evaluation? What types of supports are needed to collect the selected data? 

• Who should conduct the evaluation? 

a) If an independent evaluator is selected, then there may be a cost involved if this professional expected to 
be paid for the service. This is typically the case when contracting out the evaluation to a company which 
provides such services. However, some researchers at universities are open to in-kind evaluations that 
offer opportunities for student training and access to data for publication purposes. Thus, it may be 
useful to first approach local academic institutions for potential contacts to researchers who may be 
interested in providing the evaluation services on an in-kind or reduced fee basis. 

b) If relying on an in-house evaluator, then it is best to select someone who does not have a direct role to 
play or personal/professional investment in the program to minimize bias in the evaluation. If this is not 
possible, then an evaluation advisory committee should be formed that includes external representatives 
who can provide a degree of oversight regarding evaluation goals and procedures and offer a critical 
appraisal of the reported results. 

c) The evaluator should have sufficient knowledgeable of the problem solving court literature (e.g., general 
knowledge about these types of courts or specialized knowledge in the type of court being evaluated), 
program evaluation procedures, and data analytic methods to address the evaluation goals. If the goal is 
to only report percentages and frequency counts, then minimal statistical knowledge is required. 
However, if more advanced questions are being addressed via the evaluation, such as identifying 
statistical differences between those who complete the program and those who do not, or identifying 
predictors of program completion, then more advanced statistical knowledge is required. Involving a 
statistical consultant up front, especially when relying on in-house evaluators, can help shape the data 
collection in a manner that will be sure to address these more advanced questions. 

• What types of supports are needed to collect evaluation data? 

a) Whenever possible, it is much easier to collect data when this information is gathered as part of the 
normal procedures of the program itself — such as intake information, case plan details, administration 
of sanctions, recording of new charges, and discharge information paper-based forms or via electronic 
forms that automatically get inputted into a database. 6 If none or only some of the desired information 
is recorded in a spreadsheet or databased in real-time, then someone has to go into the records to extract 
this information. This is a time consuming task, depending on the number of case files to be reviewed 
and the type/amount of information to be extracted. This work could be the job of the evaluator or the 
evaluator’s team of research assistants, or an assigned staff member as an in-kind resource. In either 
case, it is essential that this information be recorded and extracted as accurately as possible to maintain 
the integrity of the data. Thus, training may be required to ensure that this occurs, with occasional 
monitoring to ensure fidelity to data collection protocols. 

b) Evaluators will need to ensure that they have the necessary skill set, or mentorship in the development 
of these skills, to complete the evaluation. Training also may need to be provided to staff or research 
assistants who are responsible for data recording, tracking and extraction to ensure that these 
procedures are followed correctly. 

c) Access to spreadsheet and data analytic software relevant to the specific goals of the evaluation is needed. 
For calculating of averages and frequency counts (e.g., number of admissions, average age of 
participants, most common diagnosis), programs like Microsoft Excel and Access would be sufficient. 
However, when examining outcome data, more advanced statistic methods are often required. There are 


6 If court staff members choose to record administrative data and/or other information on their court participants, we strongly encourage use 
of a Microsoft Excel spreadsheet as it would greatly aid evaluators in transferring data to statistical software programs. The most widely used 
statistical programs are able to import data directly from Excel files. Consultation with evaluators is recommended to help set up the Excel 
database to maximize the clarity and organization of the database to facilitate the data transfer. 
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a variety of programs available, such as R, SPSS, and SAS, that would meet the needs of most evaluators. 7 
When contracting out the evaluation, most evaluators would have access to these programs. When 
conducting the evaluation internally, then consideration is needed to software requirements for data 
analysis. 

d) For paper records, secure storage is required (i.e., locking filing cabinets in restricted access locations). 
For electronic records, secure storage will be required using passwords, encryption, and careful 
protocols for transportation of these e-storage devices when on portable equipment. When the 
evaluation is completed, these data will still need to be securely maintained in storage or should be 
securely shredded in the case of paper records or undergo e-destruction protocols (e.g., memory stick 
re-formatting and wiping) in the case of digital records. 

Collaborative Data Interpretation and Evaluation Feedback 

Once all data is in and has undergone analyses in accordance with the goals of the evaluation, then it is time to 
present these findings to the problem solving court team and other relevant stakeholders, which may include 
program participants and their families and funding decision-makers. Careful consideration should be given to 
whether there are any limitations or constraints on data used to evaluate the problem solving court that first needs 
to be acknowledged prior to reporting these results. Presentation of these results should be as balanced as possible, 
noting both the strengths and challenges faced by the program at an operational and implementation level, and in 
terms of its outcomes. It is helpful at this stage to invite input from knowledge users involved with the problem 
solving court who might be able to shed further light on a particular finding or add context that might explain a 
curious or unexpected result. Oftentimes, these insights re-shape the interpretation or meaning of a finding, or at 
least provide an opportunity to better understand a weakness that has been identified. After this presentation of 
findings, a written report that summarizes the evaluation procedures and its findings should be prepared. Prior to 
starting the work of the evaluation, agreements should be put in place as to how feedback will be provided, to whom, 
on what time-line, and whether these findings will be released to the public or held for internal use only. Setting 
these agreements in advance prevents confusion and missteps later that can damage or compromise the evaluator- 
collaborator relationships. 

Rather than simply present evaluation results, it is useful to the problem solving court to develop recommendations 
that can build on existing program strengths and which also provide guidance on how to work to overcome identified 
challenges with the program. Collaboratively involving the problem solving court representatives, stakeholders, and 
relevant decision-makers in this process can help evaluators provide realistic and achievable recommendations that 
work within the confines of the organization’s own constraints that is responsible for overseeing or managing the 
problem solving court. Using a collaborative process, without compromising the objectivity of the evaluation, can 
facilitate receptivity of the results by those who have diligently worked to develop, establish, and operate the problem 
solving court and/or contribute to it as a participant or in some other way. 


7 All three software packages are flexible, powerful, and capable of basic and advanced statistical analyses. Whereas SPSS requires purchasing 
yearly licences (which are very expensive), R and SAS (i.e., University Edition) are free. 
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Appendix A: Examples of Existing Problem Solving Court Evaluations 


SAMPLE EVALUATION DESCRIPTIONS OF PROBLEM SOLVING COURTS 
IN CANADA AND THE UNITED STATES 


Study title: Drug treatment court of 
Vancouver: An empirical evaluation of 
recidivism 

Author: Somers, 

Rezansoff, 

Moniruzzamen 

Publication Year: 2012 

Publication Source: Peer-reviewed journal 

Type of Court: Drug Court 

Reported Process Evaluation Information 

Information based on 400 study participants 

Client details 

Offence Information 

No specific information about the type of offences was provided. However, 87% 
of program participants had a prior sentence in the two years prior to entering 
the program. 

# discharged 

328 

# completed 

72 

Outcomes 

Use of a Control / Comparison 
Group 

Yes - Participants who completed the program were compared "non¬ 
completers" who were individuals excluded from the program after admission 
(Discharged) or who removed themselves (Withdrew) 

Criminal Justice Outcomes 

Recidivism (any convicted 
offence) in 1 st year following the 
end of their involvement in the 

program 

39% of participants re-offended (includes both completers & non-completers = 
156) 

Program graduate recidivism rate: 7% (11) 

Program non-completer recidivism rate: 93% (145) 

Drug-related offences for the program participants was significantly lower than 
the "non-completers" 

Mental Health Outcomes 

Hospitalization 

Program participants spent fewer days hospitalized for substance abuse issues 


16 I P a g e 





















Study title: Randomized 
Controlled Trial of Measure 57 
Intensive Drug Court for 
Medium- to High-Risk Property 
Offenders Process, Interviews, 
Costs, and Outcomes 

Author: Oregon 
Criminal Justice 

Commission 

NPC Research 

Publication Year: 2015 

Publication Source: Grey literature report 

Type of Court: Drug Court 

Reported Process Evaluation Information 

Information based on 388 participants (225 treatment group and 163 control group) 

(Graduates = participants who successfully completed program & Non-Graduates = Participants who did not 

graduate from program) 

Client details 

Offence Information 

Participants committed felony property or repeat drug delivery offense 

# unsuccessful, discharged 

121 

# unspecified, discharged 

22 

# active in program at time of 
study 

27 

# graduated 

49 

Average number of days in 
program 

Graduates = 448 days 

Non-graduates = 121 days 

Resource details 

Staff involved with the team 

Judge, drug court coordinator, program manager, probation officer, prosecutor, 
defense attorney, treatment representative, social services specialist, mental health 
services provider, treatment court clerk, case manager, program analyst, and 
Department of Human Services representative 

Nature of clinical and social 

services involved 

Substance abuse treatment, drug testing, provide skill development group, provide 
employment services, provide educational services, AA/ Self-help groups, assistance 
with finding employment, assistance with finding housing, assistance finding 
childcare, assistance obtaining healthcare 

Nature of Criminal Justice 

sanctions delivered 

Increased drug testing, increased drug treatment, community service, house arrest, 
or jail time 

Outcomes 

Use of a Control / 
Comparison Group 

Yes - Once placed on probation, eligible offenders were randomly assigned to 
either the drug court treatment group or a control group. The control group went 
through the traditional court processes. Treatment and control participants were 
demographically compared, and no significant differences were found between 
these two groups. 

Criminal Justice Outcomes 

Recidivism (new arrests 
during the 12-month period 
since assigned to the program 
or control group) 

28% fewer new arrests for drug court participants than the control group 

33% fewer new arrests for drug crimes for drug court participants than the control 
group 

Average # of days in jail for 
sanctions while participating 
in the program 

Graduates = 9 days 

Non-graduates = 41 days 

Average # of days in jail for 
sanctions during first year of 
program 

Graduates = 15 days 

Non-graduates = 51 days 

Mental health outcomes 

Addiction status 

There was a self- report decrease in addiction status 
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Psychosocial Outcomes (participants from both groups were interviewed when they first began the program to 
establish baseline functioning, and were interviewed again six months later to assess change in these areas) 


Employment status at 6- 
month follow-up 

Program group = 25% had jobs 

Control group = 24% had jobs 

Education at 6-month follow¬ 
up 

Program group = 76% graduated from high school or attained GED 

Control group = 64% graduated from high or attained GED 

Program group = 41% received post-high school education 

Control group = 36% received post-high school education 

Program group = 15% were currently in school 

Control group = 28% were currently in school 

Housing stability at 6-month 
follow-up 

Program group = 31% in house/apartment 

Control group = 44% in house/apartment 
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Study title: Multidimensional 
Evaluation of a Mental Health 

Court: Adherence to the Risk-Need- 
Responsivity Model 

Author: 

Campbell, Canales, 
Totten, Wei, 
Macaulay, & 
Wershler 

Publication Year: 2015 

Publication Source: Peer-reviewed journal 

Type of Court: Mental Health Court 

Reported Process Evaluation Information 

Client details 

Offence Information 

Low, medium and high risk offenders were represented in the sample, as assessed 
retrospectively from file records using the Level of Service/Risk-Need-Responsivity 
Inventory 

# Referred 

196 

# Admitted 

148 

# Non-starters (admitted, but 
did not received services) 

48 

# Partial completers 

(Admitted, but premature 
discharge) 

37 

# Successfully completed 

102 

Resource Details 

Staff involved with the team 

Judge, crown prosecutor, two duty counsels, psychiatrist, probation officer, 
mental health nurse, psychologist, and a Salvation Army representative. 

Nature of clinical and social 

services involved 

A greater number of intervention services was offered to high risk offenders, 
consistent with the Risk Principle; however, medium and low risk offenders were 
offered the same amount of services. The most common interventions offered by 
this program focused on targeting mental health factors rather than addressing 
traditional criminogenic needs. 

Nature of Criminal Justice 

sanctions delivered 

Selection and use was treatment-oriented, but specific forms of sanctions used 
was not reported 

Outcomes 

Use of a Control / 
Comparison Group 

Yes - compared participants based on their status associated with the mental 
health court. 

• Participants who completed ("completers") the program 

• Participants who did not complete ("partial-completers") the program 

• Clients who were referred but not eligible; therefore, not participating in the 
program ("nonstarters") 

Criminal Justice Outcomes 

Recidivism (new charges after 
referral to the program, based 
on an average follow-up 
period of 40 months) 

• Participants who completed the program had moderate reductions in general 
recidivism rates compared to partial-completers and nonstarters 

Mental Health Outcomes 

Symptom reduction 

Stabilized or improved for completers 

Crisis service utilization 

The largest decrease in utilization was for participants who completed the 
program 

Hospitalization (days of 
incidences) 

A small decrease across all three groups was found (completers, non-completers 
and non-starters) 

Diagnostic Status 

Participants who took part in the program had the most improved mental health 
diagnostic status of the compared groups (i.e., persistent issues, but mental health 
recovery was improving) 
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Study title: Ramsey County Mental Author: Guthman Publication Year: 2015 

Health Court Publication Source: Grey literature report 

Type of Court: Mental Health Court 

Reported Process Evaluation Information 

Client details (no specific information about completion and withdrawal rates of participants was provided) 

Offence Information 

Low, medium and high risk offences are accepted into this court 

Participants in total 

This program served a maximum of 40 participants at a time 

Resource details 

Court appearances 

Participants appear in court bi-weekly 

Staff involved with the team 

Judge, program coordinator, two community human-services case managers, 
attorney, prosecutors, probation officer 

Nature of clinical and social 

services involved 

Participants work their case managers to obtain the following services: housing, 
healthcare, [governmental] benefits, psychiatry, mental 
health care, chemical dependency treatment, therapy, 
employment, pro-social activities, and/or educational options. 

Nature of criminal justice 
sanctions delivered 

increased court appearances, assignments for court, journaling, increased 
frequency of drug testing, additional community service hours, sentence to 
service, and jail time 

Outcomes 

Use of a Control / 
Comparison Group 

Yes - individuals who met drug court eligibility criteria, but who did not 
participate in the program. This group was matched (i.e. similar) to program 
participants on age, sex, race and criminal offense characteristics. 

Criminal Justice Outcomes 

• Graduates (participants who completed the program) = followed for 3 years since leaving the program 

• Non-completers group (individuals accepted into the program, but did not complete it because of 
termination, case dismissal, or they opted out) = followed for 3 years since leaving the program 

• Comparison group (individuals who did not participate in the program) = followed for 3 years after case 
disposition 

Recidivism (any new charges 
over 3 years) 

Graduates: 30% had new charges 

Non-completers: 65% had new charges 

Comparison Group: 71% had new charges 

% of individuals who had 
spent time a period of time in 
jail during the 3-year follow¬ 
up 

Graduates: 25% in jail 

Non-completers: 84% in jail 

Comparison group: 68% in jail 

Mental Health Outcomes: only pertained to participants of the mental health court since inception in 2005 
*The program served 341 participants with serious mental illness since 2005. 

# of cases using Emergency 
Department and crisis 
services 

17 participants overall 

# of participants requiring 
hospitalization 

28 participants overall 

Medication compliance at 
graduation 

100% of overall participants at the time of graduation were compliant with 
medication 

Addiction Status at 
graduation 

100% of overall participants were addiction free at the time of graduation 
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Study title: Domestic Violence 
Courts: A Multisite Test of 

Whether and How They Change 
Offender Outcomes 

Author: Cissner, 
Labriola & Rempel 

Publication Year: 2015 

Publication Source: Peer-reviewed journal 

Type of Court: Domestic Violence Court 

Reported Process Evaluation Information 

Sample was drawn from 24 criminal Domestic Violence Courts in New York and surrounding areas 

Client details 

Offence Information 

Low, medium and high risk offenses represented 

Resource Details 

Staff involved with this 

court team 

Judge, prosecutorial team, enhanced staffing that monitored defendant compliance 
and provided assistance. 

Nature of clinical and social 

services involved 

Specific programs used for treatment are not mentioned in this article. 

Nature of CJ sanctions 

delivered 

Verbal admonishment, modify program/probation mandate, increase compliance 
monitoring, resentence to jail 

Outcomes 

Use of a Control / 
Comparison Group 

Yes - Participants in the program were compared with potential eligible clients 
whose cases were found using the statewide Order of Protection Registry. These 
comparison cases had a criminal protective order (temporary or final) issues and 
included a domestic violence-type charge. 

Criminal Justice Outcomes 

Recidivism (# re-arrested 
within 3 years of their 
conviction for the index 
offence) 

46% of program participants generally re-offended 

49% of the comparison group generally re-offended 

29% of program participated re-offended with a domestic violence charge 

32% of the comparison group re-offended with a domestic violence charge 

The average number of domestic-violence re-arrest and general re-arrests were 
significantly lower for program participants than for the comparison group. 

Outcome: no data reported for mental health and psychosocial outcomes 
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Study title: Battlefords Domestic 
Violence Treatment Options Court in 
North Battleford, Saskatchewan 

Author: Boyes 

Publication Year: 2008 

Publication Source: Grey literature report 

Type of Court: Domestic Violence Court 

Reported Process Evaluation Information 

Client details 

Offence Information 

Most common offense was assault charges 

# discharged= 

61 

# completed = 

118 

Resource details 

Staff involved with this court 

Judge, probation officer, legal aid, crown prosecutors, counsellors, victim 
services, court coordinator and aboriginal court worker. 

Nature of clinical and social 

services involved 

Provided addiction services, the Alternatives to Violence Program offered by 
Mental Health Services (provides mental health treatment programming), Prairie 
North Health Region and the Domestic Violence Program offered by 

Kanaweyimik Child and Family Services Inc. (provides programming with a 
specific emphasis on providing opportunities for clients to take part in traditional 
First Nations cultural practices) 

Outcomes 

Use of a Control / Comparison 
Group 

Yes - Participants who completed the program were compared "failed to 
complete" participants, as well as individuals who opted out of treatment. 

No adjustments were made to compensate for potential pre-group differences 
in these comparison groups that could influence outcomes. 

Criminal Justice Outcomes 

Recidivism (new domestic 
violence-related charges during 
a 36-month follow-up period 
after leaving the program 

Completers = 27% accrued new domestic violence charges 

Failed to complete = 33% accrued new domestic violence charges 

Opted out of program = 30% accrued new domestic violence charges 

% of police contacts 

Completers = 19% for domestic violence-related incidents 

Failed to complete = 34% for domestic violence-related incidents 

Opted out of treatment = 10% for domestic violence-related incidents 

Outcome: no reported data for mental health and psychosocial outcomes 


Reference Sources for Evaluations in Table Summaries: 

Boyes, M. (2008) Battlefords domestic violence treatment options court in North Battleford, Saskatchewan: Evaluation report. Calgary, 
AB. Ogden/Boyes Associates Ltd. 

Campbell, M. A., Canales, D. D., Wei, R., Totten, A. E., Macaulay, W. C., & Wershler, J. L. (20IS). Multidimensional 

evaluation of a mental health court: Adherence to the risk-need-responsivity model. Law and Human Behavior, 39, 489- 
502. doi: 10.1037/lhb0000135 

Cissner, A. B., Labriola, M., & Rempel, M. (2015). Domestic violence courts: A multisite test of whether and how they change 
offender outcomes. Violence against IJ T omen, 21, 1102-1122. doi: 10.1177/1077801215589231 

Guthmann, John H. (2015). Ramsey County mental health court: Working with community partners to improve the lives of mentally 
ill defendants, reduce recidivism, and enhance public safety. William Mitchell Law Review, 41, 948-991. 

Oregon Criminal Justice Commission, NPC Research. (2015). Randomized controlled trial of measure 57 intensive drug court for 
medium- to and high-risk property offenders: Process, interviews, costs and outcomes. 

Somers, J. M., Currie, L., Moniruzzaman, A., Eiboff, F., & Patterson, M. (2012). Drug treatment court of Vancouver: An 
empirical evaluation of recidivism. The International Journal on Drug Policy, 23, 393-400. 
doi: 10.1177/0306624X13479770 
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Appendix B: A Sample Logic Model Evaluation Framework 


Inputs 

Resources that go into 
the program 


Internal Supports 

• Court team members 
(who needs to be part 
of the team?): 

o Judge 

o Crown Attorney 
o Defence/Legal-aid 
o Mental health 
professionals, 
o Probation officers 
o Police officers 
o Community service 
providers 

o Program manager 
and/or coordinator 

• Administrative support 

• The client 

• Client's family 

• Victim representation/ 
victim services input 


Activities 

Program processes 
and procedures 


• Referral and admission 
process - screen for 
eligibility 

• Development of case 
management and 
treatment plan 

• Case conferences/ 
team meetings 

• Relationship building 
with client and/or 
family with court team 

• Mental health / drug 
screenings 

• Recidivism risk 
appraisal 

• Sanctions and incentive 
procedures 

• Court appearance 
monitoring/ check-ins 

• Training for team and 
service supports on 
issues relevant to 
program goals (e.g., risk 
appraisal and risk 
management) 


Outputs 

Result or quantification 
of the activities 


• it of people screened 

• # of people admitted / 
accepted 

• # of days from referral 
to screen to admission 
to discharge 

• # of court appearances 

• # of service 
appointments attended 

• Attendance at case 
conferences / team 
meetings 

• # of referrals made and 
accepted to community 
services and treatment 

• Nature of community 
supervision orders 

• # of graduates / 
completers 

• # of early termination 
cases (expulsion vs 
voluntary withdrawal) 

• # and nature of 
sanctions imposed 

• Reasons for non¬ 
completion 

• Client demographics 
and characteristics 
(e.g., index offences, 
criminal history, 
recidivism risk level, 
severity of mental 
health functioning or 
diagnoses/addiction 
needs, gender, age, 
ethnicity, education, 
employment, marital 
status, socioeconomic 
status) 

• # of professional 
training sessions held 


Short-term 

Outcomes 


• Lower risk of 
reoffending 

• Greater compliance 
with court-ordered 
conditions 

• Changes in targeted 
areas: e.g., decreased 
substance use, 
improved mental health 
recovery, stabilization 
of housing, improved 
education, successful 
maintenance of 
employment, financial 
stability, better 
family/marital 
relationship quality, 
reduction of 
procriminal thinking 
etc. 

• Greater use of 
community supports 

• Development of skills 
specific to the program 
targets (e.g., problem 
solving skills, life skills, 
social skills, risk 
management skills) 

• Increased personal 
accountability for 
behaviour 

• Charges withdrawn or 
stayed or reduced 


Intermediate 

Outcomes 


• Reduced criminal 
behaviour and severity 
of offending 

• Reduced victimization 

• Maintenance of mental 
health and addiction 
recovery 

• Greater community 
engagement and 
bonding 

• Reduced criminal 
justice system 
involvement 


Long-term 

Outcomes 


• Reduced criminal 
justice contacts 

• Enhanced public safety 

• Enhanced quality of life 
and well-being of target 
population 

• Development of 
prosocial and healthy 
lifestyles 

• Social stability 
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Inputs 

Resources that go into 
the program 


Activities 

Program processes 
and procedures 


Outputs 

Result or quantification 
of the activities 


Short-term 

Outcomes 


Intermediate 

Outcomes 


Long-term 

Outcomes 

External Supports 

• Community referral 
services (e.g., mental 
health, housing, 
welfare, education, 
employment, etc.) 


• Build or make use of 
existing partnerships 
with needed services 

and resources 


• # of collaborating 
partners 

• Quality of partnerships 

• Accessibility of partner 
services/supports 

• Enhanced collaboration 

and communication 
among partner 
agencies 

• Establishment of 
information sharing 
agreements and 
policies 


• Greater knowledge of 
court mandate and 

process 

• Extended partnerships 
to enhance service 
delivery 

• Greater partnership 
capacity for meeting 
community needs 

• Funding sources 


• Apply for grants or 
funding and monitor 
expenditures 


• Money received to 
support program 

• Costs of delivering 
program 


• Resource stability in the 
program 

• Cost-savings 


• Reduced costs to the 
criminal justice system 

• Changes in costs to 
social and health 

services 

• Materials, technology 
and facilities 

• Separate court room 
and/or docket 

• Maintain access to 
needed space and set¬ 
up 


• Ability to have court 
appearances and team 
meetings as expected 
and/or scheduled 


• Stability of operational 
capacity 

• Creation of physical 
identity for the 
program 

• Broader community 
and service recognition 
of the program 


• Solidification of 
program "brand" in the 
broader community 
and government 
organizations 

• Steering or Advisory 
Committee 

• Program description 
and required policies 

• Hold steering / advisory 
committee meetings 


• # of meetings held 

• # of members 
attending per meeting 


• Refinement in 
operational policies and 
procedures in the 
program 

• More efficient 
oversight of ongoing 
developments in the 
program 


• Greater consulting and 
advisory guidance to 
problem solving courts 
in other jurisdictions 

• Public relations 

• Pubic presentations 
discussing the program 

• Program website 

• Social Media contacts 

• News Media 


• # of presentations 

• # of website views 

• # of social media posts, 
shares, likes 

• # of interviews, print/ 
online articles, etc. 


• Enhanced public 
awareness of problem 
solving court and 
related supports 

• Increased public 
support for funding and 
resources for problem 
solving courts 


• Greater advocacy for 
evidence-based crime 
prevention and 
reduction strategies 



Assumptions and contextual factors that might influence program such as federal and provincial funding changes, shifts in related policies, 
competing programs in development, change in leadership and staffing supports and services 
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Appendix C: Sample Evaluation Data Collection Forms 


INTAKE/SCREENING FORM 


Client Demographic Information 8 

1. Age (in years):_ 

2. Gender: 

□ Male 

□ Female 

□ Other 

3. Ethnicity: 

□ Caucasian 

□ African Canadian / American 

□ Indigenous / Aboriginal / First Nations / Metis 

□ Latin American / Hispanic 

□ Asian / Pacific Islander 

□ indo, East, West Indian 

□ Arabian / Middle East 

□ Other (please indicate):_ 

4. Marital status at time of admission to the court: 

□ Single or never married 

□ In a relationship, not cohabitating 

□ In a relationship, cohabitating / common-law 

□ Married 

□ Divorced/Separated 

□ Widowed 

Court Referral information 


Name/ID#: 


5. Date of referral (date of first court appearance if referral date unknown):_(dd-mm-yyyy) 

6. Date formally admitted to court:_(dd-mm-yyyy) 

7. Number of days between referral and admission decision (include the date of referral and the date of 

admission in the count):_ 

8. If referred but not admitted to the court, please specify reason: 9 


8 Age, gender, ethnicity, and martial status are some of the most common demographics that should be collected. The court may also wish to 
gather information pertaining to employment status, educational status (i.e., highest level of education achieved), income, housing, health 
(i.e., physical health concerns), number of children/dependents, pregnancy status. 

9 If there are specific program eligibility criteria, then it would be best to have a checklist of all possible options in addition to an “other” 
option with space to indicate the additional reasons for non-admission. 
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9. Type and number of index criminal charges associated with referral to the court. Tick all that apply and record 
number of each type of charge next to each category. 

□ _Assault (Common, Aggravated, or Causing bodily harm) 

□ _Breach of Probation or court order (Fail to Comply/Breach of Recognizance) 

□ _Break and Enter (with and without intent) 

□ _Drug Possession 

□ _Drug Trafficking (selling)/Cultivating 

□ _Theft (includes shoplifting) 

□ _Fraud or Forgery 

□ _Mischief, Vandalism or Destruction of Property 

□ _Robbery (with or without weapon) 

□ _Weapons offence (possession of weapon, dangerous use of a weapon) 

□ _Murder/Manslaughter 

□ _Prostitution/Soliciting 

□ _Sex offence (indecent exposure, sexual interference, sexual assault, possess/make 

child porn) 

□ _Other (please specify type and number individually): 


10. Total number of criminal charges being addressed as part of referral to the court:_ 

11. Total number of times admitted to the court program (counting the current referral as 1)_ 

12. Type of community supervision order placed on client at time of referral to/during screening for court: 

□ Probation 

□ Form 12 

□ Other:_ 

13. Type of community supervision order placed on client once admitted to court: 

□ Probation 

□ Form 12 

□ Other:_ 

14. Nature of community supervision order conditions after admitted to the court (tick all that apply) 

□ Restrictions on when can be out of residence / curfew 

□ Can only be out in their residence if in the presence of a specific person(s)/professional 

□ Restrictions on being in specific geographic areas or places 

□ Restrictions on with whom the client can associate or be around 

□ Substance use restrictions/abstention 

□ Attendance of mental health counseling or other community/social services 

□ Keep the peace/be of good behaviour 

□ No access to weapons 

□ Residency conditions (required to live with certain persons and/or at certain address) 

□ Other(s):_ 
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Pre-court Admission Criminal History 


15. Prior criminal history before referral to the court. Tick all that apply since very first criminal offence 10 was 

committed excluding index offences associated with referral, and record number of each type of charge next to 
each category 

□ _Assault (Common, Aggravated, or Causing bodily harm) 

□ _Breach of Probation or court order (Fail to Comply/Breach of Recognizance) 

□ _Break and Enter (with and without intent) 

□ _Drug Possession 

□ _Drug Trafficking (selling)/Cultivating 

□ _Theft (includes shoplifting) 

□ _Fraud or Forgery 

□ _Mischief, Vandalism, or Destruction of Property 

□ _Robbery (with or without weapon) 

□ _Weapons offence (possession of weapon, dangerous use of a weapon) 

□ _Murder/Manslaughter 

□ _Prostitution/Soliciting 

□ _Sex offence (indecent exposure, sexual interference, sexual assault, possess/make 

child porn) 

□ _Other (please specify type and number individually): 


16. Total number of previous charges (excluding breaches of court orders and community supervision orders) prior 

to referral to the court:_ 

o In the 12 months prior to referral to the court:_ 

17. Total number of previous breaches court orders and community supervision orders (e.g., Form 12, probation, 

parole) prior to referral to the court:_ 

o In the 12 months prior to referral to the court:_ 

18. Has the client ever been incarcerated in a provincial jail or federal institution (excluding remand)? 

□ No 

□ Yes 

19. Total number of separate incarceration periods (excluding remand) prior to court referral:_ 

o Total number of days spent in jail/incarcerated in 12 months prior to referral:_ 

20. Has the client ever been remanded to a provincial correctional center? 

□ No 

□ Yes 

o Total number of days spent on remand in 12 months prior to referral:_ 


10 This time frame can be adjusted, such as recording criminal history only in the 12 months prior to court referral. This time frame shoidd 
be at least as long as the post-discharge follow-up period to allow for meaningful comparison of criminal records pre- and post-program 
involvement. 
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21. Recidivism risk level at time of referral/during screening/once admitted to the court. 11 

a) General recidivism risk: Very Low/Low Medium High/Very High 

i. Overall risk score:_ 

ii. Name of risk tool:_ 

b) Violent recidivism risk: Very Low/Low Medium High/Very High 

i. Overall risk score:_ 

ii. Name of risk tool:_ 

c) Sexual recidivism risk: Very Low/Low Medium High/Very High 

i. Overall risk score:_ 

ii. Name of risk tool: 


22. Identified Criminogenic Needs, directly relevant to criminal behaviour - circle all that apply 


a) Substance use 

b) Procriminal attitudes/beliefs 

c) Impulsivity 

d) Limited empathy/remorse/callousness 

e) Hostility/irritability 

f) Excessive risk taking 

g) Emotional reactivity - anger/aggression 

h) Lack of/limited structured 
leisure/recreation activities 


i) Education issues 

j) Unemployment/unstable employment 

k) Procriminal peers 

l) Few prosocial peers 

m) Family and/or intimate partner issues 

n) Psychosis-violent themes 

o) Other:_ 


23. Identified Non-Criminogenic Needs, relevant to general case management and intervention 


a) Housing/accommodations 

b) Financial need 

c) Cultural considerations 

d) Physical disabilities 

e) Intellectual disabilities 


f) Learning disabilities 

g) Anxiety 

h) Depression 

i) Psychosis - non-violent themes 

j) Other:_ 


11 Risk instruments usually have subsections that have their own separate means of representing the risk judgment (i.e., as a numerical score 
and as a structured professional judgment of risk level, severity and/or imminence of risk). Thus, these risk level options may need to be 
modified according to the risk assessment instrument used in the program or evaluation. Other useful information to record in this section 
pertains to the criminogenic and non-criminogenic needs identified as relevant to risk appraisal and case management/intervention. 
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CASE PLAN/PROGRESS FORM 


Name/ID#:_ 

24. Type of services, supports, or interventions the client received as part of their involvement in the court. Tick all 
that apply regardless of the successfulness of the services/intervention. Do not count recommended programs 
that the client never attended at all. 

□ No intervention previously received prior to admission to the court 

□ General anger management (not domestic; individually or in group) 

□ Substance abuse treatment/detox (individually or in group) 

□ Offender relapse prevention programs (individually or in group) 

□ Domestic violence/intimate partner violence programs (individually or in group) 

□ Sex offender treatment (individually or in group) 

□ Family/Couple therapy 

□ individual counseling with a mental health professional/counselor 

□ Group counseling with a mental health professional/counselor 

□ Intensive therapeutic interventions that specifically combine group and individual therapy into a 
single comprehensive intervention program (e.g., Dialectical Behaviour Therapy) 

□ Psychiatric/physician management of medication 

□ Educational upgrading 

□ Employment services or re-training 

□ Daily living services (e.g., budgeting, how to use public transportation, hygiene, cleaning) 

□ Other (include names of the program/service if uncertain of content/intervention format): 


25. Date of court appearances (dd-mm-yyyy). 


1 . 

6. 

11. 

2. 

7. 

12. 

3. 

8. 

13. 

4. 

9. 

14. 

5. 

10. 

15. 


26. Types of sanctions used: 

□ No sanctions used 

□ Attending court more often 

□ Doing community service 

□ More appointments with treatment provider 

□ More appointments with probation officer 

□ Lecture from judge 

□ Revocation of privileges 

□ Sent back to jail 

□ Other, specify:_ 
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27. Types of incentives used: 

□ No incentives used 

□ Attending court less often 

□ Fewer appointments with probation officer 

□ Positive report from treatment provider or probation officer 

□ Praise from judge or others in court 

□ Getting a gift certificate 

□ Other, specify:_ 

28. Client progress ratings. These ratings should reflect the client's progress in the program and should be made at 
each team meeting/case conference after discussing the client. Use the following scale below: 

1 = No program progress: misses most/all appointments, not motivated to change, no engagement with 

treatment providers, persistent non-compliance with intervention plans 

2 = Poor program progress: often misses appointments, appears unmotivated to change, little 

engagement with treatment providers, frequent non-compliance with intervention plans 

3 = Moderate progress: inconsistent attendance at appointments, partially motivated to change, some 

engagement with treatment providers, inconsistent compliance with intervention plans 

4 = Good progress: attends most appointments, appears motivated to change, works with treatment 

providers, good compliance with intervention plans 

5 = Excellent progress: attends all appointments, highly motivated to change, actively works with 

treatment providers, consistent compliance with intervention plans 


Date of case conference (dd-mm-yyyy) 

1 ._ 

2 ._ 

3. _ 

4. _ 

5. _ 

6 . _ 

7. _ 

8 . 


9. _ 

10 . 


Client progress rating 

1 ._ 

2 ._ 

3. _ 

4. _ 

5. _ 

6 . _ 

7. _ 

8 . 


9. _ 

10 . 
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SAMPLE DISCHARGE FORM 12 


Name/ID#: 


29. Date of discharge from the court:_(dd-mm-yyyy) 

30. Age at time of discharge (in years):_ 

31. Final court status 

□ Referred but not admitted to court before or after the screening 

□ Formally admitted to court, but did not complete the program (i.e., left before case plan was 
considered complete, left on own accord against recommendation of court team, or at the decision 
of the court due to non-compliance or new charges) 

□ Formally admitted to court and completed the program 

32. Reason for discharge from the court 

□ Successfully completed program 

□ Voluntarily withdrew 

□ Forced withdraw due to new charges 

□ Forced withdraw due to non-compliance 

□ Suspended involvement until other legal matters are addressed in another court (e.g., new charges 
resolved in traditional court before can continue in the court) 

□ Other:_ 

33. If client did not complete the program (i.e., voluntary withdraw, expelled, suspended involvement), indicate 

exact reason why:_ 

34. Total number of in-court appearances made by the client from date of referral (or first appearance) to date of 

discharge (include those dates in the count):_ 

35. New charges accrued between referral and program discharge (or 12 13 months post-referral period). Only track 
newly committed offences - exclude index offences associated with referral and offences that had dates of 
commission prior to program referral that may not have been addressed until after admission. Tick all that 
apply, and record number of offences within each ticked category of offences: 

□ _Assault (Common, Aggravated, or Causing bodily harm) 

□ _Breach of Probation or court order (Fail to Comply/Breach of Recognizance) 

□ _Break and Enter (with and without intent) 

□ _Drug Possession 

□ _Drug Trafficking (selling)/Cultivating 

□ _Theft (includes shoplifting) 

□ _Fraud or Forgery 

□ _Mischief, Vandalism or Destruction of Property 

□ _Robbery (with or without weapon) 

□ _Weapons offence (possession of weapon, dangerous use of a weapon) 


12 Please note that some of the questions on this form are based on the time period between the court referral date to discharge date for cases 
who were admitted to the court, and between the court referral date to the 12-month post-referral date for cases that were not admitted to 
the court. This time frame can be adjusted to the specific needs of the evaluation. 

13 As noted previously, the time frame for examining outcomes can vary depending on resources and time available for follow-up. A 
minimum of 12-months post-admission and 12-months post-discharge are recommended, with even longer follow-up periods being 
preferred to show long-term program impact on the measured outcomes in the evaluation. 
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□ _Murder/Manslaughter 

□ _Prostitution/Soliciting 

□ _Sex offence (indecent exposure, sexual interference, sexual assault, possess/make 

child porn) 

□ _Other (please specify type and number individually): 


36. Total number of new charges (excluding breaches of court orders and community supervision orders) earned 

between referral and discharge/12 months post-referral period:_ 

37. Total number of new charges accrued for breaches of court and community supervision orders between 

referral and discharge/12 months post-referral period (e.g., Form 12, probation, parole):_ 

38. Total number of days spent in jail/incarcerated between referral and discharge/12 months post-referral period: 


39. Overall recidivism risk level at discharge for admitted cases (or at 12 months post-referral for non-admitted 
cases) 

a) General recidivism risk: Very Low/Low Medium High/Very High 

i. Overall risk score:_ 

ii. Name of risk tool:_ 

b) Violent recidivism risk: Very Low/Low Medium High/Very High 

i. Overall risk score:_ 

ii. Name of risk tool:_ 

c) Sexual recidivism risk: Very Low/Low Medium High/Very High 

i. Overall risk score:_ 

ii. Name of risk tool:_ 


Additional Notes for Data Collection Forms: Depending on the court program, other types of information may 
be important to collect and track over time than what has been presented in these forms. To best assess change in 
the domains mentioned below, either the frequency or the client’s level of functioning over a specified period of time 
(e.g., 12-month period prior to court referral) should be tracked for all cases, as well as the time period between 
referral and discharge for admitted cases / a specified period of time (e.g., 12 months post-referral period) for non- 
admitted cases. 

Some additional information to collect may include some of the following: 

• Mental health information: history of involvement with mental health services, diagnoses at time of 
referral or admission to the court, severity of mental health issues, number of separate hospitalizations 
periods for mental health reason, number of mental health related-emergency department visits 

• Substance use information: age of first use, types of drug use, frequency of use, use of IV drugs, history 
of overdose, prior substance use treatment, prior detox services, currently in treatment, hospitalization for 
pain/injuries, types of current medication, pregnancy-related complications, relapse (include dates), 
behavioural incidents, positive urine tests 

• Psychosocial functioning: intimate partner relationship quality, family relationship quality, employment 
status, highest level of education achieved, stability of living arrangements, adequacy of financial resources 

• Medical information: current prescribed medications, list of all medical or physical conditions, history of 
head injury 


32 I P a g e 
















Appendix D: Sample Discharge/Post-Discharge Follow-up Participant Semi-Structured 
Interview 


PROBLEM SOLVING COURT SEMI-STRUCTURED 
DISCHARGE OR POST-DISCHARGE FOLLOW-UP INTERVIEW 

Introduction to the Participant: 

I understand that you have finished with the {problem solving court} program. Thank you for taking the time to sit 
with me and talk about your experiences with the program to help us evaluate it and understand what works and 
what doesn't seem to be working. 

Before we begin, I would like to ask you some questions about yourself so that we can describe the people who 
participated in these interviews as a group. 

DEMOGRAPHICS / CASE HISTORY INTERVIEW COMPONENT: 

1. Gender: Male_Female_Other_ 

2. What is your date of birth: _/_/_? 

(month) (day) (year) 

3. What ethnic group do you identify yourself with the most? 

_White/Caucasian 

_African Canadian/American 

_Indigenous/Aboriginal/First Nations/Metis 

_Latino/a / Hispanic 

_Asian 

_Middle Eastern 

_Other (please indicate):_ 

4. What is your current marital status? 

_Never married/ never been in a common-law relationship 

_Married/Common-law relationship 

_Divorced/Separated 

_Single, but dating 

_Widowed 

5. Are you currently employed? Yes _ No_ 

6. What is the highest level of education that you've achieved? 

_Elementary 

_Junior high/Middle school 

_High school/GED 

_Community College 

_University level 

7. Did you Graduate? _ Withdraw? _ Terminated? _. 
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Now I would like to ask you some questions that are specific to your involvement with the {problem solving 
court}. 

Referral and Screening Process: 

8. How did you first learn about (and who told you about) the {problem solving court} program? 

9. What were you reguired to do to get admitted into the program? 

a. What did you like about this process, and what did you not like about it? 

10. When you first learned about the program, what were you told about the possible outcomes that might 
happen to you if you participated in the program? 

a. What were you told might happen if you didn't participate in this program ? 

11. Did you have a chance to ask questions about the program before going through it? 

a. What would you have wanted to ask, or wished you had known in advance? 

Problem Solving Court Experience: 

12. What is your understanding of how the program works - tell me as if you were describing it to a friend? 

13. Tell me a bit about what being in the program was like for you? 

a. What do you remember the most about the program ? 

14. How do you see the role of each of the {problem solving court} team members involved in your case, 
including each of the following? 

a. Judge(s) 

b. Lawyers (Crown and Duty Counsel) 

c. Probation officer or Youth worker 

d. Treatment or case management staff (e.g., psychiatrist, nurse, therapist, support worker) 

15. Which team member was the most helpful for you? Why? 

16. What issues or needs were you expected to work on while involved with the program, and what help was 
provided to you (if any) to work on these things? 

a. Were the goals for your program clear/realistic/appropriate? 

17. How motivated were you to fully participate in the program when you first started it? 

a. How did this motivation change, if at all, as you moved through the program? 

18. What kinds of things were you required to do in the program ? 

a. Were there any conditions to comply with ? 

b. Were there any therapy or support case plans you were expected to follow? If so, what was 
expected of you? 

19. In what ways, if any, did being involved with the program help you work towards making these changes in 
your life? 

20. What kinds of rewards/incentives, in any, were used in the program to help you make progress? In what 
ways, in any, where these things helpful to you? 
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21. What kinds of sanctions or consequences did you experience when you were not progressing in the program 
or as compliant with it, as the team or Judge expected of you? In what ways, if any, where these things 
helpful or harmful to you? 

22. In what ways, if any, did being involved with the {problem solving court} make it harder to make positive 
changes in your life? 

23. What were the most difficult or challenging part(s) of the program for you ? 

24. What were the most helpful or rewarding part(s) of the program for you? 

Discharge Experience: 

25. How long has it been since you left the program? 

26. What do you remember about being in the courtroom that last day? 

27. How did you feel when you left the program? 

28. What are/were your expectations about what happens next for you and your legal and personal situation, 
after the program? 

Post-Discharge Experience - Program Impact: 

29. Since you left the program, how have you dealt with the issues or needs you talked about working on 
earlier in the interview? 

30. Have you been charged with any new crimes since you left the program? Yes No 

a. If yes, what types of things have you been charged with or convicted of since then? (please check 
all that apply): 

_Assault (Common, Aggravated, or Causing bodily harm) 

_Breach of Probation or court order (Fail to Comply) 

_Break and Enter (with and without intent) 

_Drug Possession 

_Drug Trafficking (selling) 

_Theft (includes shoplifting) 

_Fraud or Forgery 

_Mischief, Vandalism, or Destruction of Property 

_Robbery (with or without weapon) 

_Weapons offence (possession of a weapon, dangerous use of a weapon) 

_Murder/Manslaughter 

_Prostitution/Soliciting 

_Sexual offence (indecent exposure, sexual interference, sexual assault) 

_Other, please specify:_ 
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I just have a few more questions for you before we're finished with the interview. 


31. In what ways, if at all, do you think the program helped you avoid or reduce how often or serious you 
engaged in crime? 

32. Would you recommend this program to others? Yes No 

• If yes, why? 

• If no, why not? 

33. If you had the opportunity to talk directly to the {problem solving court} team members or the Judge(s) 
about your experience, what would you want them to know? 
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Appendix E: Sample Stakeholder Semi-Structured Interview 

Introduction to interview: In order to evaluate the [problem solving court} program, we are interested in hearing 
from people who have worked with it about what they think works well about the program and what might need to 
be changed about it to help it work better. 

Before we begin, I just have a few questions about you so that we can describe the people we interviewed as a group. 

DEMOGRAPHICS INTERVIEW COMPONENT: 

1. Gender: Male_Female_Other_ 

2. What is your date of birth: _/_/_ 

(month) (day) (year) 

3. What is your role with the {problem solving court] program? 

a. A court program team member, specify role: 

_Judge _Probation officer 

_Prosecutor _Psychiatrist 

_Defence lawyer/Duty Counsel _Psychologist 

_Case manager _Nurse 

_Social Worker _Other:_ 

b. A non-team member who provides services to program clients, specify capacity: 

_Mental health treatment provider 

_Housing support 

_Recreational support 

_Occupational therapist 

_Other:_ 

4. What is the highest level of education that you've achieved? 

_Elementary 

_Junior high/Middle school 

_High school/GED 

_Community College 

_University level 

5. What ethnic group do you identify yourself with the most? 

_White/Caucasian 

_African Canadian/American 

_Indigenous/Aboriginal/First Nations/Metis 

_Latino/a / Hispanic 

_Asian 

_Middle Eastern 

_Other (please indicate):_ 
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Now I have questions for you about various aspects of the problem solving court, some of which you may have more 
experience with then others depending on your role with the program. 

Referral and Admission Processes: 

6. What do you understand as being the goals or point of the problem solving court? 

a. To whom is it intended to provide services and supports, and what does it hope to achieve? 

7. How do potential clients of the program hear about it and get referred to it? 

a. What would you like to see stay the same or change about how people hear about the program 
and get referred to it? 

8. What do you understand about how are cases selected for admission to the program? 

a. What are reasons for not being admitted to it? 

9. Do you have any concerns with how admission decisions get made or who is included/excluded from the 
program? If yes, please tell me more about that. 

Case Management Processes: 

10. Once admitted to the program, how are clients needs evaluated or identified in the program to inform case 
management plans and goals? 

a. What do you like/dislike about this process, or find helpful/less helpful about it? 

11. How are clients monitored for their progress while in the program? 

a. What changes, if any, would you recommend to this process? 

12. What happens when a client is viewed as being non-compliance with aspects of the program? 

a. What kinds of sanctions or consequences are used to respond to this behaviour? 

b. What do you find helpful/less about how sanctions are used? Do you recommend an alternative 
means of dealing with non-compliance? 

Discharge Processes: 

13. How are decisions made about when to discharge a client from the program? 

a. Under what circumstances might a client be discharged? 

b. What concerns, if any, do you have with how these decisions are made? 

c. Any recommendations for change at this stage of the process? 

14. When a person is discharged from the program, what is your understanding of what typically happens to 
this person once they leave the program? 

a. How do they typically make out? 

b. Is anything missing from the discharge planning that would be helpful to include? 


Outcomes: 

15. What do you consider as being the most helpful aspects of the program for clients and/or for those 
impacted by the program? 

a. What do you consider to be the areas most in need of change? 
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16. In your opinion, what are the most salient outcomes for clients who participate in this program? 

a. Do you think this program helps participants reduce their involvement in criminal behaviour? 

How so? Or why not? 

b. In what ways does this program help people beyond their criminal justice issues? 

With personal needs? 

Lifestyle changes? 

Skill building? etc. 


General - Training: 

17. What kind of training, if at all, did you receive to help you do the work reguired in your role with the 
problem solving court? 

18. What kind of additional training do you think might be helpful to you or your colleagues to help the 
program operate more effectively or to help it achieve its desired outcomes? 
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Appendix F: Sample Program Participant Evaluation Informed Consent Form 


Note: This form is for use separate from program consent/agreement forms and is intended to be used 
when data collection starts near program referral (prospective design) 


You are being asked to participate in an evaluation of [(Problem Solving Court X]. The goal of this 
evaluation is to look into how well the program is run in terms of the it way operates, and also learn 
about the kind of impact it has on the people who go through this program in comparison to people who 
do not. To help us evaluate this program, we would like to use some of the information you have 
provided to the court as part of the referral and screening process for admission to it, and we would like 
to keep an eye on how you do in the program for as long as you are in it, and for ]X] months after you 
have left it. We woidcl like to do that by reviewing the records kept about you by the program, and 
^whatever other records may need to be reviewed, such as health, correctional, and police records]. We 
will also ask you to fill out a few questionnaires about yourself in terms of how you feel, think and behave 
now and again when your involvement with the program ends so that we can track any changes in these 
areas for you. In addition, we will interview you about what you thought of the program (i.e., the good 
and the bad), how it worked for you, and any thoughts you have for how to make it work better. 

Agreeing to participate in this evaluation is 100% voluntary, and you have the right to back out of this 
research at any time by letting the evaluators know your wish to do so. Your decision to helps us with 
this evaluation will have no influence on your current or future legal matters or whether you are 
admitted to the program or not. The information provided for the evaluation will be kept confidential, 
and we will only report our findings in a way that does not identify anyone who participated in the 
evaluation (e.g., no names or addresses). We will store this information securely, with access limited to 
only the evaluators. We also will not share your interview information directly with your case manager, 
the court team, or the Judge(s) involved in this program so you can feel free to speak as openly about the 
program as you would like. 

By signing this form, you are indicating that the information above has been explained to you, all your 
questions have been answered to your satisfaction, and that you understand what is being asked of you 
by participating in this evaluation. 

• Do you consent to participate in the evaluation? Yes No 

Participant/Guardian Name:_ 

Signature:_ Date:_dd/_mm/ __ yyyy 
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